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ABSTRACT 

An assessment of the Airborne Visible/Infrared Imaging 
Spectrometer (AVIRIS) performance was made for a 
flight over Mountain Pass, California, July 30, 1987. The 
flight data were reduced to reflectance using an 
empirical algorithm which compensates for solar, 
atmospheric and instrument factors. AVIRIS data in 
conjunction with surface and atmospheric measurements 
acquired concurrently were used to develop an improved 
spectral calibration. We also performed an accurate in- 
flight radiometric calibration /using the LOWTRAN 7 
radiative transfer code together with measured surface 
reflectance and atmospheric optical depths. A direct 
comparison with coincident Thematic Mapper imagery of 
Mountain Pass was used to demonstrate the high spatial 
resolution and good geometric performance of AVIRIS. 

The in-flight instrument noise was independently 
determined with two methods which showed good 
agreement. A signal-to-noise ratio was calculated using 
data from a uniform playa. This ratio was scaled to the 
AVIRIS reference radiance model, which provided a basis 
for comparison with laboratory and other in-flight 
signal-to-noise determinations. 

1.0 INTRODUCTION 

The performance characteristics of AVIRIS in the laboratory 
environment have been measured through a series of calibrations 
over the past year of instrument development (Vane et al., 1987, 
1988). The objectives of this experiment were to develop and apply 
methods for assessing the in-flight spectral, radiometric, spatial 
and signal-to-noise performance of the instrument and to provide an 
in-flight spectral and radiometric calibration. 
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AVIRIS data were acquired over the Mountain Pass region on 
the 30th of July, 1987, 10:15 AM PST, under clear sky conditions. 
Concurrent surface reflectance measurements were collected at the 
playa, asphalt and graded soil targets. These areas were selected 
for surface homogeneity, for contrast in albedo and for aerial extent 
significantly greater than an AVIRIS resolution element. A 
spectrum of the mineral bastnaesite, which occurs at Mountain Pass 
(Olson et ai., 1354) was independently measured with a laboratory 
spectrometer. Bastnaesite contains narrow spectral absorption 
features in the visible and near infrared which are suitable for 
assessing AVIRIS spectral performance. Atmospheric optical depth 
measurements were acquired contemporaneously with the over- 
flight. The coverage of the flight data and the location of the 
surface targets are shown in Figure 1. Thematic Mapper (TM) 
satellite data were obtained with coverage encompassing the 
Mountain Pass experiment region. 



Figure 1. Location of AVIRIS flight data over the Clark Mountains on July 30th, 1987 at 
10:15 AM PST. The positions of the playa, bastnaesite, asphalt and graded soil target 
areas are indicated. 


With these data the in-flight performance of AVIRIS was 
determined. Additionally, a spectral and radiometrical calibration 
was generated for these imaging spectrometer data. We describe in 
the paper the data used, the methods of analysis, and results 
obtained. 
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2.0 DATA USED IN THE STUDY 


2.1 AVIRIS Image Data. 

AVIRIS data of 2048 image lines by 614 cross-track samples 
and 224 spectral channels were acquired on an east to west flight 
line from the Ivanpah Playa across Clark Mountain to Shadow Valley. 
The data were corrected through subtraction of the end-of-line dark 
current and spatial resampling to correct for detector readout delay 
(Vane et al, 1987). These corrections generated data that were 
proportional to total radiance and spatially registered between 
spectral channels. Corrected AVIRIS spectra from the playa, the 
bastnaesite locality, the asphalt and the graded soil targets are 
plotted in Figure 2. The playa data were taken from a 200 by 200 
meter homogeneous area of Ivanpah Playa and represent an average 
of 100 AVIRIS spatial elements. The bastnaesite spectrum was 
extracted from 4 contiguous resolution elements. The asphalt target 
is a large recently surfaced parking lot from which 12 spatial 
elements were extracted and averaged. The soil target was derived 
from an average of 12 resolution elements selected from a large 
area of recently graded soil. 



WAVELENGTH (nm) 

Figure 2. AVIRIS spectra of the playa, bastnaesite, asphalt and graded soil targets with 
compensated dark current and detector readout delay. These data represent averages of 
100, 4, 12 and 12 pixels respectively. 
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2.2 Surface Reflectance Data. 


The Portable Instantaneous Display and Analysis Spectrometer 
(PI DAS) (Goetz, 1987) was used to acquire in situ surface 
reflectance spectra for the playa, asphalt and graded soil targets. 
The eight second spectral acquisition time of the instrument 
allowed the collection of 102, 93 and 35 spectra for the playa, 
asphalt and soil targets respectively in the period between 10:00 AM 
and noon. These data have a spectral sampling interval of better 
than 5 nm in the wavelength region from 450 to 2450 nm, and were 
reduced to reflectance using spectra of a pressed halon reference 
target. For comparisons and calculations using these reflectance 
spectra and the AVIRIS data, the PIDAS spectra were convolved with 
the AVIRIS channel response functions. The resulting reflectance 
spectra, reduced to AVIRIS spectral resolution, are plotted in Figure 
3. Standard deviations of 1, 1 and 2 percent reflectance were 
calculated for the playa, asphalt and soil respectively, confirming 
the nearly homogeneous surface reflectance of these targets. A 



WAVELENGTH (nm) 


Figure 3. PIDAS reflectance spectra of the asphalt, playa and graded soil target acquired 
contemporaneously with the AVIRIS flight data. The data are averages of 102, 93 and 35 
spectra and have been convolved with the AVIRIS channel spectral response functions. 
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laboratory reflectance spectrum of the mineral bastnaesite, 
acquired independently with a Beckman UV5240 spectrometer, is 
plotted in Figure 4. 



WAVELENGTH (nm) 


Figure 4. A reflectance spectrum of the mineral bastnaesite acquired with a Beckman 
UV5240 laboratory spectroradiometer showing the narrow absorption features in the 
visible and near infrared region of the spectrum caused by trivalent rare earth elements 
(Adams, 1965) 

2.3 Atmospheric Measurements. 

Solar illumination measurements were acquired with a 
Reagan-type radiometer (Shaw et al, 1973) from 7:32 am to 11:55 
am at ten minute intervals from an elevated site at the asphalt 
target on the day of the overflight. Atmospheric optical depths were 
determined from standard Langley plots, as described in Herman et 
al, (1978). The optical depths for the ten wavelength positions with 
10 nm filter bandwidths are given in Table 1. 
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Table 1. Optical depth measurements at Mountain Pass. CA 


Center Calculated 

Wavelength (nml Optical Depth 


370 

0.551 

400 

0.417 

440 

0.305 

520 

0.197 

610 

0.154 

670 

0.105 

780 

0.069 

870 

0.056 

940 

0.398 

1003 

0.047 


3.0 DATA REDUCTION 
3.1 Empirical Line Reflectance Reccvery. 

The empirical line algorithm, described for example in Conel et 
al, (1987), recovers surface reflectance from instrument response 
data. The method compensates for multiplicative components of 
solar illumination, atmospheric attenuation and instrument 
response. Additionally, the additive factors of atmospheric path 
radiance and instrument dark current are removed. 

The algorithm requires two targets of contrasting albedo and 
equivalent spectral resolution for which both surface reflectance 
and AVIRIS instrument response are known. Two first order linear 
equations are developed of the form 

DN = A * R + B 

where DN is the instrument response, R is the surface reflectance, A 
accounts for the solar, atmospheric and instrument multiplicative 
factors and B for the instrument and atmospheric additive 
components. These equations are solved for A and B for each AVIRIS 
spectral channel. The resulting equation 

R - ( DN - B ) / A 

is formed with the determined multiplicative and additive factors, 
which transform instrument DN to reflectance for the entire imaging 
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spectrometer data set. Primary assumptions of the algorithm are a 
constant instrument performance and a uniform atmosphere over the 
region to which it is applied. 

The algorithm was implemented with the reflectance spectra 
measured with PIDAS at the asphalt and graded soil targets and the 
corresponding instrument response (Figures 2 and 3). An empirical 
line spectrum derived from AVIRIS data over the bastnaesite target 
after the application of the algorithm resolves the narrow 
neodynium absorption features between 700 and 940 nm (Figure 5). 



WAVELENGTH (nm) 

Figure 5. Spectra derived from AVIRIS data using the empirical line reflectance 
retrieval algorithm. The three strong neodynium absorption features between 700 and 
900 nm are well resolved. Other features in the spectrum are discussed in the text. 

The anomalous reflectance peaks at 940 and 1125 nm are due 
to the difference in atmospheric water vapor content between the 
bastnaesite and empirical line targets. This demonstrates that the 
assumption of uniform atmosphere is not true in regions of 
atmospheric water absorption. We (Conel, et al, 1988a) have 
subsequently mapped the variations in atmospheric water vapor with 
elevation using the AVIRIS channel responses in the 940 and 1125 
nm water bands. The offsets in the reflectance curves at 
approximately 700 and 1280 nm are due to changes in the response 
functions of spectrometers A and B over the course of the flight line 
(Vane et al., 1988). The gaps at about 1400 and 1900 nm are due to 
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the strong atmospheric water bands, and the gap at about 2000 nm is 
due to a noisy detector element in spectrometer D. These features 
can be seen in most of the full spectrum plots in this paper. 

3.2 LOWTRAN 7 Radiance Model. 

In situ measurements of surface reflectance and atmospheric 
optical depth were used in conjunction with the radiative transfer 
code LOWTRAN 7 (Kneizys et al, 1988) to calculate the total 
radiance at AVIRIS for the Mountain Pass overflight. The 

atmospheric path and surface reflected as well as the total radiance 
for the Ivanpah Playa target are plotted in Figure 6. An extensive 
discussion of this methodology for modeling radiance with radiative 
transfer codes and surface and atmospheric measurements is given 
in Conel et al, (1988b). The high resolution atmospheric gas 

absorptions provide features suitable for assessment of the in- 
flight spectral performance, while modeled total radiance allows 
evaluation of the radiometric calibration. 
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Figure 6. A LOWTRAN 7 model radiance for Ivanpah Playa constrained by the conditions 
of the AVIRIS overflight. The atmospheric path, surface reflected and total radiances are 
plotted. The path radiance is a significant contributor, ranging from 80 to 10 percent of 
the total from 400 to 1300 nm. 
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4.0 ANALYSES AND RESULTS 


4.1 Spectral Performance and calibration. 

In-flight spectral performance was assessed using the 
neodymium absorption features between 700 and 940 nm and various 
atmospheric gaseous absorption features that occur throughout the 
spectral region of AVIRIS. 

To provide a common basis for comparison the laboratory 
bastnaesite spectrum was convolved with the preflight season 
channel wavelength positions and spectral response functions. 
Figure 7 shows the Gaussian approximation for the spectral response 
function for AVIRIS channel 98 with the wavelength position 
determined as the wavelength of maximum amplitude and spectral 
response characterized as the full width at half maximum amplitude 
(FWHM). The correspondence of AVIRIS spectral response to a 
Gaussian function is consistent with the instrument design and 



Wavelength [nm] 


Figure 7. A typical normalized AVIRIS spectral response function for channel 98, 
spectrometer C, compared to a Gaussian representation which closely approximates the 
measured function. 
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laboratory determination (Macenka and Chrisp 1987, and Vane et al, 
1988). In Table 2 the pre-season spectral calibration is presented 
(Vane et al, 1987) such that the spectral position of an AVIRIS 
detector is determined by the product of the detector number and the 
sampling interval summed with the starting wavelength position of 
the spectrometer. Both the convolved laboratory measured 
bastnaesite spectrum and the AVIRIS empirical line spectrum are 
plotted in Figure 8 for the wavelength region between 700 and 940 
nm. A spectral shift of several nanometers of the AVIRIS spectrum 

Table 2. Pre-season laboratory spectral calibration 


SDectrometer 

Sampling 
Int. fnrrO 

Starting 
Pos. (nm) 

Detector 

Number 

Response 
Width (nml 

A 

9.858 

396.5 

0 - 31 

9.7 

B 

9.579 

678.2 

0 - 63 

9.7 

C 

9.849 

1241.7 

0 - 63 

9.0 

D 

9.859 

1828.8 

0 - 63 

11.6 
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Figure 8. An empirical line derived bastnaesite spectrum compared to a Beckman 
spectrum of bastnaesite showing a shift of AVIRIS in-flight spectral position to shorter 
wavelengths. 
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to longer wavelengths is observed. This indicates the spectral 
channel positions for spectrometer B are shifted to shorter 
wavelengths for these flight data as compared to the laboratory 
measured bastnaesite spectrum. 

To utilize the atmospheric absorption features in an 
independent assessment of in-flight spectral performance, the 
LOWTRAN 7 total radiance was convolved with the preflight season 
AVIRIS channel spectral positions and response functions and 
compared with the AVIRIS in-flight radiance over Ivanpah Playa. 
Figure 9 shows the results for each of the four AVIRIS 

spectrometers. Spectrometer B exhibits a shift in spectral 

alignment to longer wavelengths, which is consistent with the 
determination derived from the bastnaesite spectrum. Spectrometer 
D likewise shows a shift to shorter wavelengths, while 

spectrometers A and C show good agreement between in-flight and 

laboratory spectral alignment. The predicted total radiance at 
AVIRIS allows evaluation of the in-flight spectral response. For 
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Figure 9. A comparison of AVIRIS radiance to LOWTRAN 7 modeled radiance for 
spectrometers A through D. Spectrometer B exhibits a broadened spectral response 
function and a shift to shorter wavelengths relative to the predicted LOWTRAN 7. 
Spectrometer D has also shifted several nanometers to shorter wavelengths, while 
spectrometers A and C show good agreement with the LOWTRAN 7 model. 






spectrometer B , the atmospheric oxygen feature at 760 nm is more 
completely resolved in the LOWTRAN 7 modeled data than the in- 
flight data indicating a broadening of spectral response at the time 
of flight data acquisition. This broadening of the spectral response 
function is evident in the water bands at 735, 840, 940 and 1125 nm 
as well. 

Using the LOWTRAN 7 model, an algorithm was developed to 
optimize for the Gaussian function which best describes the in- 
flight AVIRIS channel wavelength position and spectral response 
function. This algorithm minimizes the mean squared radiance 
difference between the AVIRIS retrieved radiance and modeled 
radiance while varying the response width and position of the 
Gaussian function convolved with the LOWTRAN 7 model. The 
algorithm was applied to sharp atmospheric absorption features 
where a change in position and response strongly affects the 
returned radiance. The spectral characteristics for spectrometer A 
were determined using the the water absorption band near 675 nm. 
The absorptions between 700 and 850 nm were used to fit the 
spectral characteristics for spectrometer B, while the carbon 
dioxide doublets centered at 1600 and 2050 nm were used for 
spectrometers C and D respectively. An example of the results for 
this minimizing algorithm for spectrometer C is given in Figure 10a 
and b. The spectral position is optimized for 1 nm to shorter 
wavelengths than the pre-season calibration (Figure 10a), while the 
spectral response is characterized by 1 nm less than the nominal 10 
nm FWHM response function (Figure 10b). Table 3 describes the 
resulting in-flight spectral calibration derived using this error 
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Figure 10. (a) Least squared residual fit for AVIRIS in-flight spectral position as a 
function of deviation from the pre-flight season laboratory calibration. The wavelength 
position for spectrometer C is optimized with a 1 nm shift to shorter wavelengths, (b) 
Residual minimization for spectral response function as deviation from a 10 nm 
Gaussian. Spectral response is optimal for a full width at half maximum of 9 nm. 
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minimization algorithm for each of the spectrometers. The plots in 
Figure 11 show the result of application of the in-flight spectral 
calibration to the AVIRIS flight data in comparison with the 
radiance curves obtained from LOWTRAN 7. Compared with Figure 9, 
an improvement can been seen in terms both of correspondence of 
wavelength position and resolution of the atmospheric absorption 
features. 

TABLE 3. — IrL-iliaht spectral calibration. 



Sampling 
Int. fnml 

Starting 
Pos. fnrrh 

Detector 

Number 

Response 
Width (nm) 

A 

9.858 

396.5 

0 - 31 

1 5 

B 

9.579 

674.2 

0 - 63 

1 7 

C 

9.849 

1240.7 

0 - 63 

9 

D 

9.859 

1831.8 

0 - 63 

1 0 





WAVELENGTH (nm) 

Figure 11. Results of the application of the in-flight determined spectral calibration. 
The spectral positions of spectrometers B and C are significantly improved as is the 
spectral resolution of spectrometer B. 
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4.2 Radiometric Performance and calibration. 


In Figure 9 some level of disagreement in all four 
spectrometers between the AVIRIS laboratory calibrated total 
radiance and the LOWTRAN 7 modeled radiance over Ivanpah Playa is 
observed. A ratio of these data in Figure 12 shows the LOWTRAN 7 
modeled radiance is on the average 1.2 times that of AVIRIS in 
spectrometer A, 1.4 in B, 0.85 in C and 0.80 in D. The various factors 
contributing to these discrepancies are discussed by Vane et al. 

y ■ w w y ■ 



Figure 12. Ratio of LOWTRAN 7 modeled radiance to radiance derived by applying the 
pre-flight season radiometric calibration to AVIRIS data from Ivanpah Playa. The 
deviation from a ratio of 1 indicates the amount of disagreement between the laboratory 
and in-flight radiometric response function of AVIRIS. 

Using the LOWTRAN 7 radiance model, an improved in-flight 
radiometric calibration was achieved by constructing the ratio of 
LOWTRAN 7 total radiance to AVIRIS in-flight instrument response. 
These factors for the in-flight radiometric calibration of the 
Mountain Pass data set, which transform instrument response into 
radiance, are plotted in Figure 13. This calibration provides a basis 
for analysis of these AVIRIS data requiring accurate units of total 
radiance. 


175 




WAVELENGTH (nm) 

Figure 13. Radiometric calibration factors determined for the Mountain Pass flight data. 
The coefficients provide an accurate transformation of AVIRIS instrument response in 
DN to total radiance incident at the instrument. 

4.3 Spatial Performance 

As a preliminary evaluation of spatial resolution and intra- 
image geometric quality, images of the AVIRIS and TM data were 
compared over the Mountain Pass site as shown in Figure 14 (slide 
No. 12). The AVIRIS 10-nm channels were averaged to correspond to 
the broad TM channels centered at 660, 1650 and 2200 nm; the data 
are displayed as red, green and blue respectively. The spatially 
varying geological and cultural surface features provide a basis for 
comparison of spatial resolution. AVIRIS resolves the two strands 
of Interstate 15 and the detailed geological stratigraphy to the 
south of the road which are not as clearly discernable in the TM 
image. This observed higher spatial resolution of AVIRIS is 
consistent with the designed 20 m ground instantaneous field of 
view (GIFOV) and 17 m pixel-to-pixel spacing of AVIRIS, compared 
to the 30 m GIFOV and 28.5 m pixel-to-pixel spacing of TM. The 
intra-image geometric fidelity of the AVIRIS and TM images is 
comparable. The AVIRIS image has had no geometric correction 
applied to it other than the limited on-board roll correction which is 
applied in flight (Porter and Enmark, 1987). 
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Figure 14. Comparison of AVIRIS and Thematic Mapper images of Mountain Pass, CA, at 
equivalent spectral wavelengths and bandwidths. AVIRIS more completely resolves the 
two strands of Interstate 15 as well as the fine stratigraphic units and drainage patterns 
found at the test site (refer to slide No. 12). 




4.4 Signal-to-Noise Performance. 


An in-flight assessment of the noise performance was 
developed through calculation of the standard deviation of each 
AVIRIS channel from data acquired over a nearly homogeneous region 
of the Ivanpah Playa. This analysis relates signal variation over a 
spatially homogeneous region to in-flight instrumental noise. An 
independent assessment of noise is found through determining the 
statistical variation in the dark current measured at the end of each 
image line. Results from each of these analyses are plotted in 

Figure 15. In each spectrometer, the variation calculated in the 

instrument response over the playa is similar in magnitude to the 
variation in the end-of-line dark current, establishing the dark 
current statistics as a valid estimator of instrument noise. 
Accurate determination of noise based solely on the dark current 
demonstrates AVIRIS noise for these data is independent of the 
radiance incident at the instrument. These results are consistent 
with laboratory measurements dark current and integrating sphere 
signal levels and statistics (Chrien, 1988). 
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Figure 15. Standard deviation of AVIRIS end-of-line dark current and image data over a 
homogeneous section of Ivanpah Playa. The statistical variation of the dark current 
accounts for the variation observed in the data over the playa, demonstrating dark 
current statistics to be a good estimator of instrument noise. 
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Determination of in-flight noise without requiring a large 
homogeneous surface target provides an approach for calculating the 
signal-to-noise ratio for any region of interest in the AVIRIS data. 
The ratio of average instrument response, with dark current 
subtracted, to the standard deviation of the dark current provides a 
valid estimate. 

Because the signal incident at AVIRIS is determined by the 
solar illumination, surface reflectance, atmospheric attenuation and 
atmospheric scattering, the directly calculated signal-to-noise is 
not comparable to determinations under other conditions or in the 
laboratory. This issue is resolved by scaling the actual instrument 
response to the AVIRIS reference radiance prior to calculating the 
signal-to-noise. The reference radiance is constrained by the 
conditions given in Table 4. A comparison of the AVIRIS reference 
radiance model and Ivanpah Playa radiance calculated is shown in 
Figure 16. The Ivanpah radiance values were calculated using 
LOWTRAN 7 and the PIDAS and atmospheric optical depth data as 
previously described. 

TABLE 4. 


Parameters defining the AVIRIS reference radiance. 


Radiative Transfer Code 

Atmospheric model 

Atmospheric aerosols 

Observer altitude 

Target altitude 

Target reflectance 

Date 

Hour 

Latitude 

Radiance used 


LOWTRAN 7 
Midlatitude summer 
Rural 23 km visibility 
0 km 
0 km 
0.50 

Summer solstice 
Solar noon 
45 degrees 
Reflected component 
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A signal-to-noise ratio was calculated by forming a ratio of 
image signal level over Ivanpah Playa, after dark current 
subtraction, to image noise values derived from end-of-line dark 
current statistics. To compare the in-flight signal-to-noise ratio 
for Ivanpah Playa to laboratory and other in-flight determined 
values, an AVIRIS reference radiance model was used to normalize 
the Ivanpah signal levels. Both the normalized signal levels and the 
observed signal levels were then divided by the observed noise to 
produce the signal-to-noise plots shown in Figure 17. In terms of 
the reference radiance model, spectrometer A had a peak signal-to- 
noise ratio of 90 - 100 to 1, B of 60 to 1, C of 40 - 50 to 1, and D of 
about 20 to 1 on July 30, 1987. 
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Figure 16. A plot of the AVIRIS reference radiance used to scale actual radiance falling 
on the instrument for intercomparison of signal-to-noise performance between data sets 
acquired in flight or in. the laboratory. 
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Figure 17. AVIRIS observed signal-to-noise performance over Ivanpah Playa calculated 
by a ratio of mean signal over the playa to standard deviation of the corresponding end- 
of-line dark current, and signal-to-noise scaled to the AVIRIS reference radiance. The 
scaled signal-to-noise can be compared to any other similarly scaled AVIRIS data set to 
assess stability of the radiometric response function of the instrument. 

5.0 CONCLUSION 

Through the evaluation of the in-flight performance and 
application of in-flight spectral and radiometric calibrations to the 
AVIRIS data acquired on July 30, 1987, a number of conclusions are 
reached. (1) AVIRIS successfully resolves narrow spectral 
absorption features both on the surface and in the atmosphere. (2) 
The empirical line algorithm compensates for both additive and 
multiplicative factors and directly transforms in-flight instrument 
response to reflectance. (3) Based on both surface and atmospheric 
absorption features, a shift of several nanometers in spectral 
position relative to the pre-flight season calibration is shown in 
spectrometers B and D. (4) Comparison of the modeled atmospheric 
radiance and the calibrated AVIRIS radiance show a degradation in 
spectral response function in spectrometers A and B. (5) A least 
squares residual algorithm minimizing on the LOWTRAN 7 and the 
pre-season calibrated AVIRIS radiance for both spectral position and 
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response function generates an improved spectral calibration. (6) 
In-flight modeled radiance based on concurrent in situ surface and 
atmospheric measurements predicts higher radiance in 
spectrometers A and B, with lower radiance in C and D relative to 
the pre-flight season laboratory calibration, thus demonstrating a 
corresponding change in radiometric response functions. (7) The 
modeled radiance provides a direct approach to in-flight radiometric 
calibration. (8) Comparison of coincident Thematic Mapper and 
AVIRIS imagery shows AVIRIS is resolving finer surface detail and 
shows good spatial geometry. (9) Variation in the end-of-line dark 
current is a valid estimate of noise in the AVIRIS data. 

Thus, assessment of the spectral, radiometric, spatial and 
signal-to-noise properties of AVIRIS can be achieved with a 
combination of in situ surface and atmospheric measurements in 
conjunction with a suitable radiative transfer code, providing a tool 
for sensor in-flight performance evaluation and calibration. 
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